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PEEE»ICTn/-E DISPLAY FOR A SYSTEM HAVTMG DELAYED 
FEEDBACK OF A COmiAKID ISSLiED 

PTFT.D AND BACKGROUNT i OF THK Tm/ENTIOH 

The present invention relates to a predictive display and, in particular, it 
5 concerns a predictive display for a system having delayed feedback of a command 
issued. 

By way of introduction, remotely operated devices generally require 
feedback from the device to the operator, so that the operator can decide on what 
action to take next. This is generally achieved by having a video camera or other 

10 similar apparatus disposed on the device. The video camera then relays feedback of 
the position of the device with respect to its surroundings. However, there is a time 
delay, typically of up to several seconds, between the time when the operator gives a 
movement command to the device and when the operator receives feedback with 
respect to the completion of the movement command. The delay is typically due to 

15 several factors. First, there may be a transmission time delay due to communication 
from the operator to the device and from the device to the operator. Second, there is 
a time delay due to the time taken by the device to carry out the movement 
command. Therefore, the operator needs a prediction of the final position of the 
device at the time the movement command is being created to be able to decide 

20 whether the movement command is appropriate. A solution taught by the prior art is 
to superimpose an icon such as a motion vector or arrow to represent tiie movement 
command. The beginning of the vector or arrow denotes the current position and the 
end of the vector or the arrowhead denotes the predicted position after the 
movement command has been carried out or the predicted position after a 

25 predetermined time interval. A shortcoming of the aforementioned system is due to 
the systein not being intuitive to the operator. An operator needs to pay close 
attention to the arrow or vector. A fiirther shortcoming of the aforementioned 
system is that the image of the view from the device as weU as the superimposed 
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vector are updated as the device moves, causing further conftision to the operator as 
well reduced operator performance. 

There is therefore a need for an intuitive predictive display for a system 
having delayed feedback of a command issued. 

5 SUMMARY THF. mVENTION 

The present invention is a predictive display for a system having delayed 
feedback of a command issued. 

According to the teachings of the present invention there is provided a 
method to provide feedback to an operator of a device having feedback delay, 
10 comprising the steps of: (a) displaying a first image of a view firom the device, the 
device being at a first position; (b) issuing a movement command to cause a desired 
movement of the device to a second position; and (c) displaying a second image of a 
predicted view firom the device at the second position prior to the operator receiving 

real feedback of the movement command. 
15 According to a further feature of the present invention, the second image is 

based upon at least part of the first image. 

Accordmg to a fiirther feature of the present invention, the second image 
includes a filler section outside of the at least part of the first unage. 

According to a further feature of the present invention, the filler section 

20 includes a pattern. 

According to a further feature of the present invention, the filler section 

includes a repetitive pattern. 

According to a fiirther feature of the present invention, the filler section 
includes historic image data of the predicted view. 
25 According to a further feature of the present invention, there is also provided 

the step of displaying a third image of an actual view firom the device at the second 
position. 
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According to a further feature of the present invention, there is also provided 
the step of limiting the movement command to ensure that the second image can be 
based-upon at least part of the first image. 

According to a further feature of the present invention, the step of issuing the 
5 movement command and the step of displaying the second image, occur 
substantially at the same time. 

According to a further feature of the present invention, the step of displaying 
the first image is perfoimed by displaying the first image on a screen, the screen 
having a fi^e disposed thereon, the first image beuig disposed substantially within 
10 the firame and wherein the step of displaying the second image is performed by 
displaying the second image on the screen such that, the second image includes 
substantially all image elements of the first image. 

According to the teachings of the present invention there is also provided, a 
feedback system for an operator of a device having a camera, comprising: (a) a 
15 control arrangement configured for issuing a movement command to cause a desired 
movement of the device from a first position to a second position; and (b) a display 
configured for: (i) displaying a first image of a view firom the device, the device 
being at a first position; and(u) displaying a second image prior to the operator 
receiving real feedback of the movement command, the second image being a 
20 predicted view firom the device at the secondposition, the second image being based 
upon at least part of the furst image. 

According to a further feature of the present invention, the display is fiirther 
configured for displaying a third image of an actual view firom the device at the 
secondposition. 



PCT/IL2004/000680 

WO 2005/010639 



4 

BRIEF DESCFTTnTQM OF T FTF. DRAWINGS 

The invention is herein described, byway of example only, witli reference to 
the accompanying drawings, wherein: 

Fig. 1 is a schematic view of a remotely operated system having a display 
5 that is constructed and operable in accordance with a preferred embodiment of the 
present invention; 

Fig. 2 is a view of the display of Fig. 1 showing an unage of a predicted view 
from the device due to a movement command; 

Fig. 3 is a view of the display of Fig. 1 showing an image of a predicted view 
10 from the device due to a movement command, the image including a repetitive 
pattern "filler" section that is constructed and operable in accordance with a first 
altemate embodiment of the present inventioiu 

Fig. 4 is a view of the display of Fig. 1 showing an image of the actual view 
from die device after the movement command has been completed and the image of 
15 the view from a camera at the new position has been received; 

Fig. 5 is a view of the display of Fig. 1 showing the image of a predicted 
view from the device due to a forward or zoom-m movement command; 

Fig. 6 is a view of a display prior to issuing a movement command that is 
constructed and operable in accordance with a second alternate embodunent of the 

20 present invention; 

Fig. 7 is a view of the display of Fig. 6 showmg an image of a predicted view 
from the device due to a movement command; and 

Fig. 8 is a view of the display of Fig. 6 showing an image of the actual view 
from the device after the movement command has been completed and the image of 
25 Review from a camera at the new position has been received. 
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DESCRIPTTQl^T OF THE PRHFEREED EK>iBODB4ENTS 



The present invention is a.predictive display for a predictive display for a 
system having delayed feedback of a command issued. 

The principles and operation of a predictive display for a system having 
5 delayed feedback of a command issued according to the present invention may be 
better understood with reference to the drawings and the accompanying description. 

Reference is now made to Fig. 1, which is a view of a remotely operated 
system 10 having a display 12 that is constructed and operable in accordance with a 
preferred embodiment of the present invention. Remotely operated system 10 also 
10 has a remotely controlled device 14 and an operator module 22. Device 14 has 
disposed thereon, a camera 18 and a transceiver 20. Device 14 is t^^ically a 
remotely operated vehicle or surveillance camera arrangement It will be apparent to 
those skilled in the art that when device 14 is a surveiUance camera, movement of 
device 14 is represented by movements and zooming of camera 18. However, it will 
15 be apparent to those skiUed in the art that device 14 could be any device requiring 
operator control via a predictive display. Additionally, it will be appreciated by 
those ordinarily skilled in the art that remotely operated system 10 includes systems 
whereby the operator is disposed within or on top of device 14. Operator module 22 
includes display 12, a movement controller 16, a processor 24 and a transceiver 26. 
20 Camera 18 is configured to send images of views from device 14 via transceiver 20 
to transceiver 26. Transceiver 20 and transceiver 26 tj^pically define a wireless 
communication link. However, it will be apparent to those skilled in the art that 
processor 24 and device 14 may be connected using a communication cable or other 
communication link. Images received by transceiver 26 are processed by 
25 processor 24 and then displayed by display 12. The frequency of images received by 
transceiver 26 is tj^pically frequent enough, such that, the view from camera 18 is 
viewed by the operator as a "continuous" video or a series of discrete images. The 
operator issues movement commands to device 14 using movement controller 16. 
Movement controUer 16 is typically includes a joystick or pointing device such as a 
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mouse or a steering wheel as well as an other movement control devices capable of 
instructing regarding direction and steering as well as zooming a lens of camera IS. 
.. JSiovement commands are sent via processor 24, transceiver 26 and transceiver 20 to 
an actuator 27 which is configured to move device 14. As cam^a IS is generally 
5 disposed in a known relationship with respect to de\dce 14, images of views 
captured by camera 18 define the position and orientation of device 14 in relation to 
the surroundings. Display 12 shows an image 28 of a view from device 14 while 
device 14 is at a first position. It should be noted that image 28 is updated constantly 
to reflect views captured by camera 18 of the first position, taking into account the 
10 delay between camera 18 capturing the view and predictive display 12 displaying 
image 28. 

By way of introduction, due to the time delay between the operator issuing a 
movement command to cause a desired movement of device 14 from the first 
position to a second position and the operator receiving feedback of that movement 

15 command, the operator needs a way to visualize the fiaial position due to the 
movement command at the time the movement command is issued. The desired 
movement may be linear, rotational or a more complex movement Therefore, 
according to the present invention, a predicted image of the view from device 14 at 
the second position is displayed on display 12 typically at the time the operator 

20 issues the movement command, which is prior to the operator receiving real 
feedback of the actuated movement command. Real feedback is defined herein as an 
actual image of a view from camera 18 at the second position and not a predicted 
image of a view from camera 18 at the second position. 

Reference is now made to Fig. 2, which is a view of display 12 showing an 

25 image 30 of a predicted view from device 14 due to a movement command. 
Remotely operated system 10 is configured such that image 30 of the predicted view 
from device 14 at the second position is displayed on display 12 at substantially the 
same time that a movement command is issued. Therefore, image 30 is displayed 
prior to the opearator receiving real feedback of the actuated movement command. 

30 Image 30 is based upon at least part of knage 28 (Fig. 1). It v^dll be appreciated by 
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those skilled in the art that as movement controUer 16 is actuated, the image 
displayed on predictive display 12 is altered to reflect the current predicted view 
from camera IS after the movement command is complete. The current predicted 
view from camera 18 after the movement command is complete, is formed by 

5 manipulating image 28, for example by moving or scaling or rotating image 28, as 
appropriate, such that, the center of an image of a view from camera 18 at the 
second position corresponds to the center of predictive display 12. Scaling of 
image 28 is necessary where forward and reverse movement commands are issued. 
Other manipulation of image 28 is performed to reflect the desired results. Scaling 

10 of image 28 is Ulustrated with reference to Fig. 5. Therefore, as movement 
conlroUer 16 is actuated, the im^e displayed by predictive display 12 gradually 
changes from image 28 to image 30. Therefore, there is a plurality of images which 
are displayed on predictive display 12, which represent the actuating of movement 
controUer 16 in order to move device 14 from the first position to the second 

15 position. In tlie example of Fig. 2, the movement command given by the operator to 
move device 14 from the fu^t position to the second position was to turn device 14 
in a left direction. Therefore, in this example, the right portion of image 30 is 
formed by using most of image 28 except for part of the right portion of image 28. 
The left portion of image 30 has a blank "fiUer" section 32 indicative of image data 

20 which cannot be derived from image 28. It should be noted that as the operator 
moves the steering control of movement controller 16 to the left, image 30 will 
change to include gradually less of the right portion of image 28 and more of "filler" 
section 32. It wUl be appreciated by those skilled in the art that other operations 
including lateral, rotational, scaling and other image manipulations may be 

25 performed by processor 24 on image 28 to create unage 30 depending on the desired 
movement command. Additionally, image 30 may be created by processor 24 using 
advanced image processing techniques to produce an image which is based upon 
image 28 as well as other images of prior views stored by remotely operated 
system 10. It should be noted that unage 30 is created by altering image 28 to reflect 

30 the predicted view from camera 18 when device 14 is at the second position and not 
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by superimposing an arrow or vector on image 2S to shovv where the second 
position is. Tj'pically, processor 24 instructs actuator 27 to automalicaUy execute the 
movement commands as steering occurs. OptionaUy, the movement command is 
manuaUy confirmed to actuator 27 of device 14 to commence actual execution of 
5 ihe movement command. Actuator 27 then completes the desired movement of 
device 14 to the second position. Although image 30 continually reflects an image 
of the predicted view at the second position, image 30 is continually updated by 
views captured by camera 18 even while camera 18 is moving. 

As an optional feature, the movement command is limited to ensure that 
10 image 30 can be based upon at least part of image 28 so that the operator will be 
able to visualize at least partially the effect of the movement command. If the 
movement command is such that the second position cannot be described, using 
image 28 then image 30 will be blank unless other historic data is available. 

Reference is now made to Fig. 3, which is a view of display 12 showing an 
15 image 34 of a predicted view from device 14 due to a movement command. 
Image 34 includes a "filler" section 36 that is constructed and operable in 
accordance with a first alternate embodiment of the present invention. Filler 
section 36 is typically a static pattern which is manipulated in the same way as 
image 34, for example, by translation, rotation or scaling. FiUer section 36 is 
20 generally a repetitive pattern such as a grid. OptionaUy, additional historic image 
data of the predicted view is used to fill in the fiUer section of image 34 instead of 
using a blank filler or a repetitive pattern filler. The historic image data is based on 
previous images at or close to the second position. 

Reference is now made to Fig. 4, which is a view of display 12 showing an 
25 image 38 of the actual view from device 14 after the movement command has been 
completed and the image of the view from camera 18 at the second position have 
been received by operator module 22. It should be noted that image 38 is formed by 
gradual changes to image 30 as camera 18 captures views as camera 18 moves. 

Reference is now made to Fig. 5, which is a view of display 12 showing an 
30 image 40 of a predicted view from device 14 due to a forward or zoom-in movement 
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command. It is seen that image 40 is an enlarged version of image 23 (Fig. 1). 
Therefore, image 40 reflects the forward or soom-in movement command, which 
brings the objects of image 28 closer. 

Reference is now made to Fig. 6 and Fig. 7. Fig. 6 is a view of a display 42 
prior to issuing a movement command that is constructed and operable in 
accordance with a second altemate embodiment of the present invention. Fig. 7 is a 
view of display 42 showing an image 44 of a predicted view from device 14 due to a 
movement command. Display 42 has a centraUy disposed frame 46. Frame 46 is 
either an image or a permanent fixture which is disposed on display 42. Before a 
movement command is issued, an image 48 of an actual view from device 14 is 
displayed wholly within frame 46. At this point, device 14 is at a first position. The 
operator then issues a movement command, for example, to move device 14> to the 
left to a second position. As the movement command is issued, image 44 is 
displayed on display 42. Image 44 is image 48 which is moved to the right on 
15 display 42 and partially outside of fiame 46 to reflect the steering to the left by the 
operator. However, substantially all image elements of image 48 are included within 
image 44. The term "substantially all unage elements are included" is defined herein 
to include, for example, when some of the details of image 48 are lost due to scaling 
of image 48 to produce image 44. 
20 Reference is now made to Fig. 8, which is a view of predictive display 42 

showing an image 50 of the actual view from device 14 after the movement 
command has been completed and the image of the view frohi camera 18 at the 
second position has been received. 

It will be appreciated by persons skiUed in the art that the present invention is 
25 not limited to what has been particularly shown and described hereinabove. Rather, 
the scope of the present invention includes both combinations and sub-combinations 
of the various features described hereinabove, as well as variations and 
modifications thereof that are not in the prior art which would occur to persons 
skilled in the art upon reading the foregoing description. 



